DEVOPS-7803 restore log stream #2972
Conversation
Re-establish connection while the pod is still running
|
|
||
| func (s *restoreLogStreamReader) Read(p []byte) (n int, err error) { | ||
| n, err = s.reader.Read(p) | ||
| defer func() { s.lastReadTime = metav1.Now() }() |
There was a problem hiding this comment.
This looks like we take the timestamp after the stream close. My main concern in #2903 (comment) was that if there was a log at the time of the stream close we might lose some records.
Did you have a way to reproduce the failure so we can have at least empirical proof that this approach works?
There was a problem hiding this comment.
Yes, there is a way to reproduce it, and I did it. Unfortunately, each test takes 4 hours as I just run an action from a blueprint that uses a long sleep and echo.
This approach in manual tests always resulted in a duplicated last line of the log, which was independent of the frequency of the log. The last line was repeated even if it was the only line since 4 hours.
There was a problem hiding this comment.
Right. Should we then add some deduplication for that case (if we can)?
There was a problem hiding this comment.
Deduplication would be quite an overhead for what we are trying to achieve. This problem manifests only after 4 hours, and a single duplicated line for tasks that take over 4 hours, in my opinion, doesn't justify the need to add deduplication logic. This logic would require storing the last read line in memory and comparing it after reconnecting. Additionally, we should not assume that our users' logs distinguish between lines, which would complicate such a simple comparison.
Therefore, in my opinion, we should accept this single duplicated line that occurs on rare occasions.
|
@e-sumin, any updates on the review? |
Change Overview
Unfortunately, there is a hardcoded timeout in the kubelet server. If we want to support long-lasting phases, we need to work around it.
This PR introduces re-establishing the log stream connection while the pod is still running.
Pull request type
Please check the type of change your PR introduces:
Issues
Test Plan
I created a phase executing function
KubeTask, with a long-lasting command (sleep). I ensured that the logs were read until the end of its execution and that the output was properly gathered.